A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

Recently, Deep Neural Networks (DNNs) have recorded great success in handling medical and other complex classification tasks. However, as the sizes of a DNN model available dataset increase, training process becomes more computationally intensive, which usually takes longer time to complete. In this work, we proposed generic full end-to-end hybrid parallelization approach combining both data parallelism for efficiently distributed scalable models. We also Genetic Algorithm based heuristic resources allocation mechanism (GABRA) optimal distribution partitions on GPUs computing performance optimization. applied our real use case 3D Residual Attention Network (3D-ResAttNet) efficient Alzheimer Disease (AD) diagnosis multiple GPUs. The experimental evaluation shows that is scalable, achieves almost linear speedup with little or no differences accuracy when compared existing non-parallel